Validation of an academic listening test: Effects of "breakdown" tests and test takers' cognitive awareness of listening processes

This study investigated the breakdown effect of a listening comprehension test, whether test takers are affected in comprehending lectures by impediments, and collected test takers’ cognitive awareness on test tasks which contain listening breakdown factors how they perceived these impediments. In this context of the study, a “Breakdown” is a test task that contains impeding factors which could limit test takers’ listening comprehension. Impeding factors used in this study are British accent, fast speech rate, and noise. The listening comprehension test contained four different types of lectures with two topics: dinosaurs and weather changes. In total, ninety-six test takers took the test and each group of twenty-four test takers were assigned to following test groups: the “regular”, the “British”, the “speech rate”, and the “noise” groups. The name of each group represented the source of interference or impediment in the listening task. Along with test takers’ performance on listening test, the design of the study was informed by test takers’ responses to a questionnaire on their cognitive awareness of listening, metacognitive process, and test-taking strategy uses. Both quantitative and qualitative analyses were used on test scores and responses on questionnaire. To estimate validity and reliability of the test, several quantitative analyses were conducted. Cronbach’s alphas and confidence intervals of alphas were calculated for each test and the coefficient values were observed to vary among the tests. A Generalizability study and an analysis of variance (ANOVA) were conducted to measure the breakdown effect on the test. The result of generalizability study showed that variance components for interaction between person and items were larger than variance components for person, which indicated confounding effect. However, the analysis of variance showed that breakdown effect on test was not detected, but the effect of talk types was significantly different. Classical item analyses were also iii conducted to observe differences on item functions among four tests. A few items were flagged as easy items, however difficulty and discrimination indices were varied among groups. To understand test takers’ perspectives on second language listening, both quantitative and qualitative analyses were conducted. Test takers’ responses on questionnaire showed test takers from different groups demonstrated similar metacognitive strategy uses, monitoring behaviors, and test-taking strategies. Findings showed that there was no intentional breakdown effect on all groups. Low reliability coefficient and generalizability analysis showed that there was insignificant evidence to support breakdown effect of three impediments on test takers’ listening proficiency. However, one of impediments, noise functioned as an impediment in the test. Some test takers from the noise group reported that their comprehension was impeded by noise. At the test level, talk type was identified as breakdown factor from both quantitative and qualitative analyses. At item level, regardless of item difficulty indices, lower item discrimination indices indicated that items functioned differently when an impediment was added in the test. Listening passages functioned as a more influential factor in comprehension than the breakdown factors. Test takers who already had some background knowledge could perform better than those who do not have the knowledge. Furthermore, test takers with higher scores were able to use various types of metacognitive process than those with lower scores regardless of groups. This study suggests that several considerations are needed to make authentic listening tasks when listening impediments are included in the second language test development such as degrees of impediments, threshold of the breakdown factor, and breakdown effects on both item iv and test levels. Moreover, metacognitive process embedded listening items could contribute to understand second language learners’ listening comprehension